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eBay manages ... 


Over 212,000,000 registered users 
Over 1 Billion photos 














— eBay users worldwide trade more than $1590 
worth of goods every second 

— eBay averages over 1 billion page views per 
day 

— At any given time, there are approximately 
105 million listings on the site 

— eBay stores over 2 Petabytes of data — over 
200 times the size of the Library of Congress! 

— The eBay platform handles 3 billion API calls 
per month 











+ In a dynamic environment 


— 300+ features per quarter 


— We roll 100,000+ lines of code every two weeks 








Over % Million pounds of A ®8&ctgoun 
Kimchi are sold every year! an et company 











- In 33 countries, in seven languages, 24x7 














>26 Billion SQL executions/day! 
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212M Users 


300+ Features = 99.94% 
Per Quarter bé: 


100K LOC/Wk 6M LOC 


e Our site is our product. We change it incrementally through implementing new features. 
e Very predictable development process — trains leave on-time at regular intervals (weekly). 
e Parallel development process with significant output -- 100,000 LOC per release. 


e Always on — over 99.94% available. 


All while supporting a 24x7 environment 
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Enable seamless growth 


Deliver quality functionality at 
accelerating rates 


Enable rapid business innovation 
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+ Scale Out, Not Up 
— Horizontal scaling at every tier. 
— Functional decomposition. 


Prefer Asynchronous Integration 
— Minimize availability coupling. 
— Improve scaling options. 

+ Virtualize Components 


— Reduce physical dependencies. 
— Improve deployment flexibility. 


« Design for Failure 
— Automated failure detection and notification. 
— “Limp mode” operation of business features. 
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Registered Users ALAN 
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eBay architecture versions 
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e Built over a weekend in Pierre Omidyar's living room in 1995 

e System hardware was made up of parts that could be bought at Fry's 
e Every item was a separate file, generated by a Perl script 

e No search functionality, only category browsing 


calculus.ebay.com trig.ebay.com thompson.ebay.com thomson.ebay.com 


FreeBSD 


FreeBSD 


FreeBSD FreeBSD 








"cgi" "listings" 
This system maxed out at 50,000 active items — -ud 
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e 3-tiered conceptual architecture (separation of bus/pres and db access tiers) 
e 2-tiered physical implementation (no application server) 
e C++ Library (eBaylSAPI.dll) running on IIS on Windows 


e Microsoft index server used for search 
e Items migrated from GDBM to an Oracle database on Solaris 


iquana.ebay.com komodo.ebay.com cayman.ebay.com gecko.ebay.com 
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python.ebay.com 





e Servers grouped into pools (small soldiers) 

e Resonate used for front end load balancing and failover 

e Search functionality moved to the Thunderstone indexing system 

e Back-end Oracle database server scaled vertically to a larger machine (Sun E10000) 
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e Second Database added for failover 
e CGI pools, Listings, Pages, and Search continued to scale horizontally 


However ... 


By November 1999, the database servers approached their limits of physical growth. 


Resonate Resonate Resonate Resonate 


Solaris 
Oracle Oracle 


bull.ebay.com bear.ebay.com 
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e Database "split" technology. 
e Logically partition database into separate instances. 
e Horizontal scalability through 2000, but not beyond. 





O 
Listings Pages 





Solaris Solaris Solaris Solaris 





Oracle Oracle Oracle Oracle 


bull. ebay.com bear.ebay.com chard cab/bongo 
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e Horizontal scalability through database splits 


e Items split by category 
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+ Application Server 
— Monolithic 2-tier Architecture 
— 3.3 Million Line C++ ISAPI DLL (150MB binary) 
— Hundreds of developers, all working on the same code 
— Hitting compiler limits on number of methods per class (!!) 
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+  Re-wrote the entire application in J2EE application server framework 
— Gave us a chance to architect the code for reuse and separation of duties 


+ Leveraged the MSXML framework for the presentation layer 
— Minimizing the development cost for migration 


+ © Implemented a development kernel as a foundation for programmers 
— Allowed for rapid training and deployment of new engineers 
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Scaling the Data Tier 








+ Spread the Load 
— Segmentation by function. 
— Horizontal splits within functions. 


e Minimize the Work 
— Limit in database work 


+ The Tricks to Scaling 
— How to survive without transactions. 
— Creating alternate database structures. 
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+ Segment databases into functional areas 
— User hosts 

— Item hosts 

— Account hosts 

— Feedback hosts 

— Transaction hosts 

— And about 70 more functional categories 


+ Rationale 


— Partitions data by different scaling / usage characteristics 
— Supports functional decoupling and isolation 
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° Split databases horizontally by primary access path. 


+ Different patterns for different use cases 
— Write Master/Read Slaves 
— Segmentation by data; Two approaches 
+ Modulo on a key, typically the primary key. 
Simple data location if you know the key 
Not so simple if you don't. 
° Map to data location 
Supports multiple keys. 
Doubles reads required to locate data. 


SPOF elimination on map structure is complex. 


+ Rationale 
— Horizontal scaling of transactional load. 
— Segment business impact on database outage. 
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e Separate Application notion of a database from physical implementation 
e Databases may be combined and separated with no code changes 
e Reduce cost of creating multiple environments (Dev, QA, ...) 


Application Servers 
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+ No business logic in database 
— No stored procedures 
— Only very simple triggers (default value population) 


+ Move CPU-intensive work to applications 
— Referential Integrity 
— Joins 
— Sorting 


+ Extensive use of prepared statements and bind variables 
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Auto-commit for vast majority of DB writes 


Absolutely no client side transactions 
— Single database transactions managed through anonymous PL/SQL blocks. 
— No distributed transactions. 


How do we pull it off? 
— Careful ordering of DB operations 
— Recovery through 
e Asynchronous recovery events 
+ Reconciliation batch 
* Failover to async flow 


Rationale 
— Avoid deadlocks 
— Avoid coupling availability 
— Update concurrency 
— Seamless handling of splits 
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Scaling the Application Tier 
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e Spread the Load 
— Segmentation by function. 
— Horizontal load-balancing within functions. 


e Minimize dependencies 
— Between applications 
— Between functional areas 
— From applications to data tier resources 


e Virtualize data access 
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+ Step 1 - Throw out most of J2EE 
— eBay scales on servlets and a rewritten connection pool. 


° Step 2 — Keep Application Tier Completely Stateless 
— No session state in application tier 
— Transient state maintained in cookie or scratch database 


* Step 3— Cache Where Possible 


— Cache common metadata across requests, with sophisticated cache refresh 
procedures 


— Cache reload from local storage 
— Cache request data in ThreadLocal 
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e Strictly partition application into tiers 
Presentation 
Business 
Integration 


CEE NN 
Presentation Tier - 
Command (View) 
AO/AOF (View) 
Business Tier BO/BOF | usiness Logic 
Yy v 


Integration Tier DO/DAO Data Access Layer (DAL) 


XML Model 
Building Logic 
B 


<M > 
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+ What is the DAL? 
— eBay’s internally-developed pure Java OR mapping solution. 


— All CRUD (Create Read Update Delete) operations are performed 
through DAL’s abstraction of the data. 


— Enables horizontal scaling of the Data tier without application code 
changes 


« Dynamic Data Routing abstracts application developers from 
— Database splits 
— Logical / Physical Hosts 
— Markdown 
— Graceful degradation 


« Extensive JDBC Prepared Statements cached by DataSources 
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e Partition code into functional areas 
e Application is specific to a single area (Selling, Buying, etc.) 
e Domain contains common business logic across Applications 


e Restrict inter-dependencies 
e Applications depend on Domains, not on other Applications 
e No dependencies among shared Domains 


Shared 
Domains 
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e Segment functions into separate application pools 
e Minimizes / isolates DB dependencies 
e Allows for parallel development, deployment, and monitoring 


Viewltem Pool SYI Pool 
http://cgi.ebay.com ... http://cgi5.ebay.com ... 


Load Balancing 


IIS WebServers 


AppServers 


Load Balancing 





Load Balancing 


m 7 eb 


© 2006 eBay Inc. 


Scaling the Application Tier — Platform Decoupling 
EEE 2 


Domain Partitioning for Deployment 
Decouple non-transactional domains from transactional flows 
Search and billing domains are not reguired in transaction processing. 
Fraud domain is reguired but easier to manage as separate deployment. 


Integrate with a combination of asynchronous EDA and synchronous SOA 
patterns. 
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Scaling Search 
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+ In 2002, eBay search had reached its limits 
— Cost of scaling third-party search engine had become prohibitive 
— 9 hours to update the index 
— Running on largest systems vendor sold — and still not keeping up 


+ eBay has unique search requirements 
— Real-time updates 
+ Update item on any change (list, bid, sale, etc.) 
+ Users expect changes to be visible immediately 
— Exhaustive recall 
« Sellers notice if search results miss any item 
+ Search results require data (“histograms”) from every matching item 
— Flexible data storage 
* Keywords 
* Structured categories and attributes 


+ No off-the-shelf product met these needs 
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« Real-time feeder infrastructure 
— Reliable multicast from primary database to search nodes 


+ Real-time indexing 
— Search nodes update index in real time from messages 


+ In-memory search index 


+ Horizontal segmentation 
— Search index divided into N slices (“columns”) 
— Each slice is replicated to M instances (“rows”) 


— Aggregator parallelizes query over all N slices, load-balances over M 
instances 


e Caching 
— Cache results for highly expensive and frequently used queries 
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Scaling Operations 
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+- Demanding Requirements 
— Entire site rolled every 2 weeks 
All deployments require staged rollout with immediate rollback if necessary. 
More than 100 WAR configurations. 
— Dependencies exist between pools during some deployment operations. 
— More than 15,000 instances across eight physical data centers. 


Rollout Plan 
— Custom application that works from dependencies provided by projects. 
— Creates transitive closure of dependencies. 
— Generates rollout plan for Turbo Roller. 


+ Automated Rollout Tool (“Turbo Roller”) 
— Manages full deployment cycle onto all application servers. 
— Executes rollout plan. 
— Built in checkpoints during rollout, including approvals. 
— Optimized rollback, including full rollback of dependent pools. 
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+ Centralized Activity Logging (CAL) 
— Transaction oriented logging per application server 


+ Transaction boundary starts at request. Nested transactions supported. 


« Detailed logging of all application activity, especially database and other external 
resources. 


* Application generated information and exceptions can be reported. 
— Logging streams gathered and broadcast on a message bus. 
* Subscriber to log to files (1.5TB/day) 
* Subscriber to capture exceptions and generate operational alerts. 
* Subscriber for real time application state monitoring. 
— Extensive Reporting 
+ Reports on transactions (page and database) per pool. 
+ Relationships between URL’s and external resources. 
+ Inverted relationships between databases and pools/URL’s. 


e Data cube reporting on several key metrics available in near real time. 
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Enabling seamless growth 


e Massive Database and Code Scalability 


Delivering quality functionality at 
accelerating rates 


+ Further streamline and optimize the eBay 
development model 


Enabling rapid business innovation 
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